Papers with diagnostic method
To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. (2022.coling-1)
Copied to clipboard
| Challenge: | Reasoning and knowledge-related skills are considered as fundamental skills for natural language understanding (NLU) tasks. |
| Approach: | They propose a method to diagnose correlations between an NLU dataset and a specific skill. |
| Outcome: | The proposed method is able to diagnose correlations between dataset and logical reasoning skill on 8 MRC and 3 NLI datasets. |
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on diagnosing LMs' reasoning abilities in natural language understanding tasks. |
| Approach: | They propose a diagnostic method for first-order logic reasoning with a proposed benchmark, LogicNLI. |
| Outcome: | The proposed method disentangles the target FOL reasoning from commonsense inference and can be used to diagnose LMs from four perspectives: accuracy, robustness, generalization, and interpretability. |
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. |
| Approach: | They propose a diagnostic method based on cross-modal input ablation to assess the extent to which pretrained models integrate cross-module information. |
| Outcome: | The proposed method evaluates the model's performance on the other modality based on inputs from one or both modality. |